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METHOD AND APPARATUS TO ENCRYPT VIDEO DATA STREAMS 

The present invention relates to flie field of data encryption; more specifically, it 
relates to encrypting of video data for subsequent rendering on processor-based video 
systems. 

With the increasing prospects for widespread use of multi-media communications 
through open networks, such as the Internet and wireless networks, the need for 
confidentially and privacy as well as controlled access will become increasingly important. 
Encryption of data sent over these networks has become the solution of choice. 

However, as broadband contents increase, encryption at the content or service 
provider end and especially decryption time at the user end is either slow Oow performance 
processor) or expensive (high performance processor) because of the burden put on the 
processors. The latest methods of encryptmg based on video frames helps somewhat, but 
video flames stUl require encrypting very large amounts of data that will only increase as 
broadband content increases. 

A first aspect of the present invention is a method of encrypting a video data 
stream, llie video data stream partitioned into units based upon a type of data contained 
within the units comprising: determining for each unit the type of data contained within the 
unit; and encrypting a particular unit or a portion of the particular unit based upon the type 
of data contained within the unit. 

A second aspect of the present invention is a method of encrypting a video data 
stream, the video data stream partitioned into NAL units formed from partitioned slices, 
each NAL unit containing either header data, intra data or inter data, comprising: 
determining for each NAL unit whether the NAL unit contains header data, intra data or 
inter data; and encrypting a particular NAL unit or a portion of the particular NAL unit 
based upon whether the particular NAL unit contains header data, intra data or inter data. 

A third aspect of the present invention is s system for encrypting a video data 
stream, the video data stream partitioned into units based upon a type of data contained 
within the units comprising: means for determining for each unit flie type of data contained 
within the unit; and means for enciypting a particular unit or a portion of the particular unit 
based upon the type of data contained witiiin the unit. 
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A foimh aspect of the present invention is a system of encrypting a video data 
stream, the video data stream partitioned into NAL units formed ftom partitioned sUces. 
each NAL unit containing either header data, intra data or inter data, comprising: means for 
determining for each NAL unit whether the NAL unit contains header data, intra data or 
inter data; and means for encrypting a particular NAL unit or a portion of the particular 
NAL unit based upon whether the particular NAL unit contains header data, intra data or 
inter data. 

The features of the invention are set forth in the appended claims. The invention 
itself, however, will be best understood by reference to the following detailed description 
of an iUustrative embodiment when read in conjunction with the accompanying drawings. 
15 wherein: 

no. 1 is an illustration of data grouping before partitioning; 
HG. 2 is an illustration of the formation of data partitions from data groups; 
HGs. 3 A and 3B are illustrations of a RTP/NAL (network abstraction layer) unit 
packages; 

20 FIG. 4 is an illustration of the field structure of NAL units; 

FIG. 5 is a schematic block diagram of a system for encrypting the International 
Telecommunications Union Telecommunications Standardization Sector (TW-T) H.264 
video data stream according to the present invention; and 

FIG. 6 is a flowchart of the method steps for encrypting video data according to the 
25 present invention. 

FIGs. 1 through 3 A and 4 are provided as an aid to understanding the present 
invention and merely illustrate the ITU-T H.264 standard digital data stream stmcture. 
FIG. 3B extends the invention to a situation not presently defined in ITU-T H.264 

FIG. 1 is an illustration of data grouping before partitioning. A slice is defined as 

30 an integer number of macro-blocks ordered contiguously in raster scan order within a 
particular slice group, which may not be contiguous within the picture. In FIG. 1 a slice 
includes a slice header field, a header data field, an intra data field and a inter data field. 
The index "i" is used to indicate the specified data corresponds to the i* macro-block in the 
slice. Header data includes the macro-block type (syntax = mb_typeOi). Macro block 

35 types include I blocks, P blocks, B blocks, SI blocks and SP blocks, each of which has sub 
macro-block types not of interest to the present invention. 
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An I block is defined as a block coded using prediction (estimation of the value 
being decoded) ftom decoded samples witiiin the same block. An SI block is defined as a 
switching I block. A P block is defined as a block coded using prediction fiom previously 
decoded reference pictures. A SP block is defined as a switching P block. A B block is 
defined as a predictive block. There are five predictive modes for B blocks, list 0, list 1 . 
bi-predictive, direct and intra predictive. I and SI blocks are intra predictive blocks 
because the prediction is derived firom decoded samples of the current decoded picture. P, 
SP and B blocks are inter predictive blocks because the prediction is derived from decoded 
samples other than the current decoded picture. Note the definition relating to I, P, B, SI 
and SP blocks are appUcable to macro-blocks, flames, fields and pictures bearing the same 
designations, however in the case of macro-blocks it should be understood that different 
types of macro-blocks can exist within a single slice of a single picture. Moreover, even 
sub-blocks of a macro-block can be of different types. 

The intra data field contains coded intra block (i. e. I and SI blocks) data. The inter 
data field contains coded mter block (i. e. P, SP and B block) data. 

FIG. 2 is an iUustration of the formation of data partition types from data groups. 
Partitioning is defined as the division of a set (i. e. the elements of the slice of FIG. 1) into 
subsets (i.e. the elements of the partition types of FIG. 2) such that each element of the set 
is m exactly one of the subsets. In FIG. 2. the sUce illustrated in FIG. 1 is partitioned into 
three partition types. Partition type A includes a slice header field (syntax = 
slice_headerO), a slice ID field (syntax = slicejd), a header data field and a trailing bits 
field (syntax = tb). The content of the slice header field of partition type A is the content 
of the slice header field of the slice illustrated in FIG. 1. The slice ID field is a new field 
(relative to FIG. 1), which indicates which slice the partition is derived from. The contents 
of the partition type A header data field is the contents of the data header field of the slice 
illustrated in FIG. 1. The traiUng bits field is a new field (relative to FIG. 1) and is used to 
make the number of bits in partition type A an even multiple of 8. 

Partition type B includes the slice ID field described supra, an intra data field and a 
trailing bits field. The content of the partition type B intra data field is the content of the 
intra data field of the slice illustrated in FIG. 1. The trailing bits field is again used to 
make the number of bits m partition type B an even multiple of 8. 
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Partition type C includes the slice ID field described supra, an inter data field and i 
trailing bits field. The content of the partition type C inter data field is the content of the 
inter data field ofthe slice illustrated in HG. 1. TTie trailing bits field is again used to 
make the number of bits in partition type C an even multiple of 8. 

HGs. 3A and 3B are illustrations of a RTP/NAL unit packages. The ITU-T H 264 
standard specifies a NAL unit as a generic format for use in both packet orientated and bit- 
stream systems. A NAL unit is constmcted by concatenating raw byte sequence payloads 
(RBPS). In the case of partitioned data, each RBPS may contain only one partition type. 
For flie purpose ofthe present invention, the NAL units are illustrated as having been 
encoded in an exemplary transmission layer using real time protocol (RTP). Other 
protocols such as MPEG-2 Transport, MPEG-2 Program Stream and H.233 may also be 



used. 



In FIG. 3 A, an RTP packet stream includes an RTP header and a single NAL unit. 
TTie RTP header (or packetized elementary stream (PES) headers for MPEG-2) conveys 
infonnation about the encryption method. The NAL unit includes an NAL header (see 
definition infta) and a RBSP payload. TTie RBSP packet ofthe NAL unit may contain 
partition type A data, partition type B data or partition type C data. 

In FIG. 3B, an RTP packet stream includes an RTP header and multiple NAL units. 
The first NAL miit (NAL unit 1) contams information about the encryption method. Each 
NAL unit includes an NAL header (see definition infra) and RBSP payloads. The RSBP 
packet of NAL miit 1 contains supplemental enhancement information (SEI) information 
(syntax = reserved_SEI_message). Reserved_SEI_message includes information about the 
encryption of NAL units 2 through N The format of reserved_SEI_message must be 
agreed upon by both sender and receiver, so the receiver knows how to interpret the SEI 
message. TTie RBSP packet of NAL unit 2 contains partition type A data, the RBSP packet 
of NAL unit 3 contains partition type B data and the RBSP packet of NAL unit 4 contains 
partition type C data. Any NAL unit 2 through N may contain a partition type A RBSP, a 
partition type B RBSP or a partition type C RBSP, but only one. 

no. 4 is an illustration ofthe field structure of a NAL unit. In FIG. 4. a NAL unit 
includes a NAL header and a RBSP packet, which is a partition type A RBSP packet. The 
NAL header is defined as the group of fields forbidden_bit. nal_storage_idc and 
nal_unit_type. The nal_umt_type indicates whether the unit contains data for an A. B or C 
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type partition. H.264 defines a hexadecimal value of nal_umt_type = 0x2 indicates an A 
partition type. 0x3 indicates an B partition type and 0x3 indicates an C partition lype Other 
fields in tiie header are as illustrated. The RBSP packet contains a shoe header field 
(syntax = sUce_header), a sUce ID field (syntax = sUceJd). a slice data field (syntax = 
sUce_data) and a trailing bits field (syntax = ttailingLbits). The slice header field is 
included only when the NAL unit contains a partition type A RBSP. Partition type B and 
C RBSPs contain only die sUce ED field, flie slice data field and the tiaUing bits field. ITie 
slice data field contains header, intra or inter data as discussed supra. 

The sUce header includes several fields, the most relevant to the present invention 
being a fi^e number field (syntax = fi^e_number), a picture structure field (syntax = 
picture_structure) and a slice type field (syntax = slice_type Jdc). The picture structure 
field indicates if the data is field data or flame data. A frame is defined as containing 
sampled and quantized luma and chroma data of all rows of a picture. A frame consists of 
two fields, a top field and a bottom field. A field is defined as an assembly of alternate 
rows of a fiame. The sKce type field indicates if the slice is a P, B, I, SP or SI slice. 

FIG. 5 is a schematic block diagram of a system for encrypting the ITU-T H.264 
video data stream according to the present invention. In FIG. 5, an encryption device 100 
includes a H.264 encoder 105, an analyzer 1 10, a control interface 115, an encryption 
contix>ller 120. a switch 125. encryptors 130A, 130B and 130C and key generators 135A, 
135B and 135C. 

H.264 encoder 105 receives input video data stream 140 and generates compressed 
video data sh:eam 145. Compressed video data stream 145 is formatted in NAL units, each 
of which incorporates one of either an A type partition, a B type partition or a C type 
partition as illustated in FIGs. 3 and 4 and describe supra. Analyzer 1 10 analyzes 
compressed video data stream 145 by reading the NAL headers to obtain, for example, 
coding information as to tiie type of partition (A, B, C) the NAL unit contains, or storage of 
the corresponding picture in the reference picture buffer. The collected information is 
passed to encryption controller 120 via a statistics signal 150. Encryption controller 120 
compares the statistics on each NAL unit to a set of selection and encryption rules 
generated by contirol interface 115, and selects which NAL units will be encrypted and how 
they will be encrypted via an enciyptor conti-ol signal 155 sent to switch 125 and a key 
selection signal 160 sent to key generators 135A, 135B and 135C. 



5 



wo 2004/056112 



PCT/IB2003/00596S 



10 



15 



20 



Selection and enciyption rules may be global (i.e. partition based) wherein the NAL 
values of unit parameters nal_miit_type and slice_type_idc define what type of partition to 
encrypt or selection and encryption rules may be local (i.e. based on attributes other than 
partition type). A local selection and encryption rule must always have a global selection 
and encryption rule associated with it Local selection rules allow only selected NAL units 
of the globaUy selected partition type to be selected and encrypted. Local selection and 
encryption rules may be based on any non-partition type related field in the NAL unit. For 
example, local selection and encryption rules may be based on the number of bits in the 
slice data field (syntax = slice_data). 

Control mterface 1 15 can implement a fixed set of selection and encryption rules or 
a programmable set of selection and enciyption rules for enciyption controller 120 to apply 
to the mformation about a particular NAL unit obtained from statistics signal 1 50. 
Programmable rules allow the user to dynamically adjust the selection rules, possibly 
taking into account information external to video data stream 140. 

The selected enciyptor (either encryptor 130A, 130B or 130C) enciypts the entire 
NAL unit or a portion of the NAL unit. For example, the NAL header or one or more 
fields within the NAL header, the RBPS field or one or more sub-fields within the RBSP 
field (for example the slice data field) or just selected groups of bits with the NAL unit 
may be encrypted. When the NAL unit header is encrypted, the corresponding RBSP is 
not be encrypted, thus saving enciyption time. If an RBSP is encrypted, the corresponding 
NAL unit header is not encrypted and the NAL unit header conveys information needed for 
decryption of the RBSP. For example, the sender and receiver agree upon an encryption 
method for a particular partition type and the partition type is described in the NAL header 
field nal_unit_type. 

Sunilarly, enciyption information may be contained in the NAL header or one or 
3 0 more fields within the NAL header, the RBPS field or one or more sub-fields within the 
RBSP field. The example of the reserved_SEI_message field of the RBSP packet was 
illustrated in FIG. SB and described supra. Ahnost any other fields of the NAL unit may 
be used (for example, the trailing_bits field) by "misusing" those fields. 

The output of switch 125 is a selectively encrypted video data signal 165. 
35 Three enciyptors 130A, 130B. and 130C are illustrated in FIG. 5, In a first 

exemplary implementation, each enciyptor 130A. 130B and 130C is respectively dedicated 



25 



wo 2004/056112 



PCT/ro2003/005965 



to a different partition type, i. e. A type, B type or C type. In a second ex6nq)lary 
implementation, each enciyptor 130A, 130B and 130C is dedicated to a different type of 
encryption method in both the generic sense and the specific sense. Examples of generic 
encryption methods include variable key, fixed key, single encryption, double encryption 
methods. In the case of double encryption, two encryptors would be cascaded withm one 
of enciyptors 130A. 130B or 130C. Examples of common specific encryption mefliods 
include the Data Encryption Standard (DBS), flie triple DBS (3DES), the Advanced 
Encryption standard (AES) and the Digital Video Broadcast - Common Scrambling 
Algorithm (DVB-CSA). 

Shnilarly, each encryptor 130A, 130B or 130C may be supplied with its own 
respective key generator 135A. 135B or 1350 or each key generator may be available to 
each encryptor. There may be more or less than three encryptors, there may be more or 
less than three key generators and the number of encryptors need not be the same as the 
number of key generators. Table 1 lists several examples of encryption policy, the key 
NAL unit parameter and flie rationale and benefit of that policy. 

TABLE I 



Partitions 
encrypted . 


Policy 

Partitions Encryption 
not encrypted method 


NALunit 


Benefit 


BandC 


A any 


nal_unit_type 


Enable analysis of 
headers 


A 


B and C any 


nal_unit_type 


Protection with least 
effort (i.e. software) 


A 

BandC 


Variable key 
Fixed key 


naljinit_type 


Unequal protection 



A Double encrypt nal_unit_type Unequal protection 

B and C Single encrypt 
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A BandC any nal_umt_type Protecting only I or 

slicejype_idc SP slices 

When data partitioning is used, the important low-level data in a packet is 
concentrated in certain partitions rather than being mixed with other data and scattered 
throughout the packet. Hence, by choosing to encrypt a certain partition in a packet and by 
which encryption method, a certam level of protection can be obtained. For example, 
encrypting the high level information (e. g. partition type A) will make the whole packet 
practically undecodable, while encrypting lower level information (e. g. partition types B 
and C), flie packet may be decoded, but at a lower quality. 

Different strategies are conceivable for implementing this principle. These 
strategies can take into account size and significance of partitions, depending on the 
application. For example, when encodmg video with the intention to distribute it in band 
width-limited or error prone environments such as the Internet or ad-hoc wireless 
networks, a higher number of intra macro-blocks can be dehberately used to reduce the risk 
or error propagation. (As defined supra, intra macro-block can be decoded mdependently 
and is not used for decoding inter macro-blocks.) In such cases, it is usefiil to encrypt the 
partitions containing mtra data (e.g. partition type B), i.e. I and SI frames, even though 
such partitions can contain more bits than other partitions. Another example is encryption 
of partitions encompassing mter data (e.g. partition type C) in inter coded frames, i. e. P, B, 
and SP frames. 

FIG. 6 is a flowchart of the method steps for encrypting video data according to the 
present invention. In step 170, video data is grouped into slices as illustrated in FIG. 1 and 
described supra. In step 175, the grouped video data is partitioned into A type partitions, B 
type partitions and C type partitions as illustrated in FIG. 2 and described supra. In step 
180, the partitioned data is encoded according to ITU-T H.264 standards as illustrated m 
FIGs. 3 and 4 and described supra. In step 1 85, a NAL unit is selected and its partition 
type (A, B or C) determined based on the parameter nal_unit_lype in the NAL header of all 
NAL units or alternatively based on the parameter nal_unit_type and the parameter 
slice_lype_idc found in the slice header field of NAL units containing partition type A 
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RBSPs. In step 190, it is detennined whether or not to encrypt a particular NAL unit based 
on selection and encryption rules as discussed supra in reference to FIG. 5. If the NAL 
unit is not to be encrypted, then the method loops to step 185 and the next NAL unit in the 
data stream is selected. If the NAL unit is to be encrypted, then the method proceeds to 
step 195. In step 195, the encryption method and encryption key are selected and in step 
200 the NAL unit or portion of the NAL unit is encrypted. The method then loops to step 
185 where the next NAL unit is selected. 
The description of the embodiments of Ae present invention is given above for the 
understanding of the present mvention. It will be understood that the invention is not 
limited to the particular embodiments described herein, but is capable of various 
15 modifications, rearrangements and substitutions as will now become apparent to those 
skilled in the art without departing from the scope of the invention. Therefore, it is 
intended that the following claims cover all such modifications and changes as fefl within 
the true spirit and scope of the invention. 
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